with virtual object(s) in a mixed reality space including a virtual space and real space 



comprising: 

viewpoint detection program step for detecting the location/posture of a 
viewpoint of the operator; 

geometric information acquisition program step for acquiring geometric 
information of real object(s); 

recognition program step for recognizing a current relative relationship 
between the virtual object (s) and real omect(s); 

a rule memory for storing rules for controlling the action of the virtual 

object(s); 

computation program /step for determining the next action of the virtual 
object(s) in accordance with the rulds stored in said rule memory and in correspondence 
with the location/posture of the real object(s), and computing the location/posture of the 
virtual object(s) after the determined action; and 

presentation program step for generating at least one image of the virtual 
object on the basis of the locationvlposture of the virtual object(s) after the action and the 
location/posture of the viewpoint position of the operator, and for representing the mixed 
reality space to the operator by superimposing the image(s) of the virtual object(s) on the 
operator's view of the real space. 



REMARKS 

Applicants request favorable reconsideration and allowance of this 
application in view of the foregoing amendments and the following remarks. 
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Claims 1-39 are pending in this application, with Claims 1, 20, and 39 being 

independent. 

Claims 1-9, 11-18, 20-28, 30-37, and 39 have been amended. Applicants 
submit that support for these amendments can be found in the original disclosure, and 
therefore no new matter has been added. 

Claims 2, 3, 5-7, 14, 15, and 20-38 were rejected under 35 U.S.C. §112, 
second paragraph, as allegedly being indefinite. The claims have been amended in light of 
the Examiner's comments, and Applicants submit that the Examiner's objections have 
been overcome. Favorable reconsideration and withdrawal of the rejection are sought. 

Claims 1-11, 14, 15, 17-19, 20-30, 33, 34, and 36-39 were rejected under 35 
U.S.C. §102 as being anticipated by U.S. Patent No. 5,577,981 (Jarvik) . Claims 12, 13, 16, 
31, 32, and 35 were rejected under 35 U.S.C. §103 as being obvious over Jarvik . 
Applicants respectfully traverse these rejections for the reasons discussed below. 

As recited in independent Claim 1, the present invention relates to a 
simulator apparatus having, inter alia, computation means for determining a next action of 
virtual object(s) in accordance with rules stored in a rule memory and in correspondence 
with the location/posture of real object(s), and computing the location/posture of the virtual 
object(s) after the determined action. Through this feature, virtual object(s) are not only 
controlled to change in response to an operator's operation, but also the next action of the 
virtual object(s) is controlled in accordance with both the rules stored in a rule memory 
and the location/posture of real object(s). 

Applicants submit that Jarvik fails to disclose or suggest at least this feature. 
That patent is essentially directed to a virtual reality exercise machine for a single user. 
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The exercise machine varies a virtual image to be presented to an operator based on the 



movement/posture of the operator and an operation by the operator.l However, it does not 



vary a virtual object based on a combination of a stored rule and the location/posture of a 
real object. / 



Accordingly, Applicants submit that Claim 1 is patentable over Jarvik . 
Claims 20 and 39 recite similar features and are believed allowable based on similar 
distinctions over Jarvik . 

For the foregoing reasons, Applicants submit that this application is in 
condition for allowance. Favorable reconsideration, withdrawal of the outstanding 
rejections, and an early Notice of Allowance are requested. 

Applicants' undersigned attorney may be reached in our 
Washington, D.C. office by telephone at (202) 530-1010. All correspondence should 
continue to be directed to our below-listed address. 




Respectfully submitted, 




Attorney for Applicants 
Brian L. Klock 
Registration No. 36,570 



FTTZPATRICK, CELLA, HARPER & SCINTO 
30 Rockefeller Plaza 
New York, New York 10112-3801 
Facsimile: (212)218-2200 



BLKVcmv 
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VERSION WITH MARKINGS TO SHOW CHANGES MADE TO THE CI^IMS ^ 

"a ^ 

% 4. ^ 

1. (Amended) A simulator [game] apparatus with which an operator [a playetjj ^"^ 
/s a simulation [game] with virtual object(s) in mixed reality space including a virtual space^ 
and real space, said simulator comprising: 

viewpoint detection means for detecting the location/posture of a viewpoint of the 
operator [player]; 

geometric information acquisition means for acquiring geometric information of 

real object(s); 

recognition means for recognizing a current[,] relative relationship between the 
virtual object(s) and real object(s); 

a rule memory for storing rules for controlling the action of the virtual object(s); 

computation means for determining the next action of the virtual object(s) in 
accordance with the rules stored in said rule memory and in correspondence with the 
location/posture of the real object(s), and computing the [thee] location/posture of the virtual 
object(s) after the determined action; and 

presentation means for generating at least one [of] image of the virtual object(s) 
on the basis of the location/posture of the virtual object(s) after the determined action and the 
location/posture of the viewpoint position of the operator [player], and for representing the mixed 
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reality space to the operator [player] by superimposing the image(s) of the virtual obiect(s) 
[image(s)] on the operator's [player's] view of the real space. 

2. (Amended) The [A game] apparatus according to claim 1, wherein said 
presentation means further comprises [comprising]: 

image-capturing means for capturing real space images of said operator's [player'] 
view of real space images of said player's view of the real space; 

image generation means for generating mixed reality images representing [of] the 
mixed reality space by superimposing or overlaying said image(s) of the virtual object(sl 
[image(s)] on said real space images; and 

a video see-through type display means that the operator wears [player wares] 
wherein said mixed reality images are displayed. 

3. (Amended) The [A game] apparatus according to claim 1, wherein said 
presentation means further comprises an [comprising, An] optical see-through type display 
means that the operator wears [player wares] wherein said virtual object image(s) are displayed. 

4. (Amended) The [A game] apparatus according to claim 1, further comprising, 
status detecting means for detecting status of the operator [player]; 

wherein said computation means determines a next action of the virtual object in 
accordance with the rule stored in said rule memory and in correspondence with the 
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location/posture of the real object and/or the status of the operator , and computing a 
location/posture of the virtual object after the determined action. 



5. (Amended) The apparatus according to claim 1, wherein the current [,] relative 
relationship includes a layout relationship between the virtual object and real object at the current 
[that] time in the mixed reality space. 

6. (Amended) The apparatus according to claim 1, wherein the current [,] relative 
relationship includes a behavior of the real object with respect to the virtual object at the current 
[that] time in the mixed reality space. 

7. (Amended) The apparatus according to claim 1, wherein the real object 
includes the operator [player] himself or herself, and said recognition means recognizes a current 
[,] relative relationship between the virtual object and the operator [player]. 



8. (Amended) The apparatus according o claim 1, wherein the real object includes 
a plurality of operators [players] who operate said simulator [game] apparatus, and the plurality 
of operators [players] share a single mixed reality space. 



9. (Amended) The apparatus according to claim 1, wherein the real object is an 
object which is fixed in position, and 
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said geometric information acquisition means comprises: 

a predetermined memory for pre-storing location information and shape 

information of the real object; and 

means for reading out the location information and shape information of the real 

object from said predetermined memory as needed. 

11. (Amended) The apparatus according to claim 1, wherein the real object is an 
operator [a player], and 

said geometric information acquisition means comprises: 

a sensor for detecting a location/posture of a head of the operator [player]; 

and 

means for setting a region having a fixed, known shape that approximates 
the operator [player] in the mixed reality space in accordance with the detected 
location/posture of the operator's head. 



12. (Amended) The apparatus according to claim 1, wherein when the simulation 
[game] is a battle simulation [game] with the virtual object, an objective is to decrease an 
expected score of the operator [player]. 
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13. (Amended) The apparatus according to claim 1, wherein when the simulation 
[game] is a cooperative simulation [game] with the virtual object, an objective is to increase an 
expected score of the simulation [player]. 



14. (Amended) The apparatus according to claim 1, wherein one of the rules 
stored in said rule memory controls the action of the virtual object on the basis of an objective of 
the simulation [game] and a relative relationship between the virtual object and real object. 

15. (Amended) The apparatus according to claim 1, wherein one of the rules 
stored in said rule memory expresses the action of the virtual object as an action pattern with a 
predetermined aim for achieving an [the] objective of the simulation . 



16. (Amended) The apparatus according to claim 15, wherein the pattern has a 
path disadvantageous to the operator [player] in consideration of a layout relationship between 
the virtual object and real object. 



17. (Amended) The apparatus according to claim 11, wherein when the operator 
[player] is one of the real objects, an output from said viewpoint detection means for detecting 
the location/posture of the viewpoint of the operator [player] is also used as information which is 
to be acquired by said geometric information acquisition means and pertains to a location and 
shape of the operator [player]. 
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18. (Amended) The apparatus according to claim 1, wherein said viewpoint 
detection means detects a location/posture of the [a] head of the operator [player], and 

said apparatus further comprises detection means for detecting a location/posture 
of a hand of the operator [player]; and 

means for recognizing a relative location of the hand of the operator [player] with 
respect to the head as a command on the basis of an output from said detection means. 

20. (Amended) An image processing method for a simulator [game] apparatus 
with which an operator [a player] plays a game with virtual object(s) in a mixed reality space 
comprising: 

viewpoint detection step for detecting the location/posture of a viewpoint of the 
operator [player]; 

geometric information acquisition step for acquiring geometric information of real 

object(s); 

recognition step for recognizing a current [,] relative relationship between the 
virtual object(s) and real object(s); 

[a rule memory for storing rules for controlling the action of the virtual object(s);] 
computation step for determining the next action of the virtual object(s) in 
accordance with the rules stored in a [said] rule memorv , which stores rules for controlHng the 
action of the virtual obiectfs), and in correspondence with the location/posture of the real 
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object(s), and computing the [thee] location/posture of the virtual object(s) after the determined 
action; and 

presentation step for generating at least one [of] image of the virtual object(s) on 
the basis of the location/posture of the virtual object(s) after the determined action and the 
location/posture of the viewpoint position of the operator [player], and for representing the mixed 
reality space to the operator [player] by superimposing the image(s) of virtual object(s) 
[image(s)] on the operator's [player's] view of the real space. 

21. (Amended) The method according to claim 20, wherein the operator wears 
[player wares] a video see-through type display, and said presentation step further comprising, 

image-capturing step for capturing real space images of said operator's [player's] 
view of the real space; 

image generation step for generating mixed reality images representing of the 
mixed reality space by superimposing or overlaying said image(s) of virtual object(s} [image(s)] 
on said real space images and for displaying said mixed reality images on the display. 

22. (Amended) The method according claim 20, wherein the operator wears 
[player wares] an optical see-through type display and said presentation step representing the 
mixed reality space to the operator [player] by displaying the image(s) of virtual object(s) 
[image(s)] on the display. 
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23. (Amended) The method according to claim 20, further comprising, 
status detecting step for detecting status of the operator [player]; 
wherein said computation step determines a next action of the virtual object in 
accordance with the rule stored in said rule memory and in correspondence with the 
location/posture of the real object and/or the status of the operator , and computing a 
location/posture of the virtual object after the determined action. 



24. (Amended) The method according to claim 20, wherein the recognition step 
recognizes the current[,] relative relationship including [includes] a layout relationship between 
the virtual object and real object at the current [that] time in the mixed reality space. 

25. (Amended) The method according to claim 20, wherein the recognition step 
recognizes the current[,] relative relationship including [includes] a behavior of the real object 
with respect to the virtual object at the current [that] time in the mixed reality space. 



26. (Amended) The method according to claim 20, wherein [the real object 
includes the player himself or herself, and] the recognition step includes the step of recognizing a 
current[,] relative relationship between the virtual object and the [player.] operator, and the real 
object includes the operator himself or herself. 
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27. (Amended) The method according to claim 20, wherein [the real object 
includes a plurality of players who operate the game apparatus, and the] a plurality of operators 
[players] share a single mixed reality space and the real obiect(s) used in the computation step 
includes a plurality of operators who operate the apparatus . 

28. (Amended) The method according to claim 20, wherein the real object is an 
object which is fixed in position, and 

the geometric information acquisition step includes the steps of: 
pre-storing location information and shape information of the real object in a 

predetermined memory; and 

reading out the location information and shape information of the real object from 

the predetermined memory as needed. 

30. (Amended) The method according to claim 20, wherein the real object is an 

operator [a player], and 

the geometric information acquisition step includes the steps of: 
detecting a location/posture of the [a] head of the operator [player]; and 
setting a region having a fixed, known shape that approximates the operator 

[player] in the mixed reality space in accordance with the detected location/posture of the head. 
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31. (Amended) The method according to claim 20. wherein when the simulation 
[game] is a battle simulation [game] with the virtual object, an objective used in said 
computation step is to decrease an expected score of the operator [player]. 

32. (Amended) The method according to claim 20, wherein when the simulation 
[game] is a cooperative simulation [game] with the virtual object, an objective used in said 
computation step is to increase an expected score of the operator [player]. 



33. (Amended) The method according to claim 20, wherein one of the rules 
controls the action of the virtual object on the basis of an objective of the simulation [game] and 
a relative relationship between the virtual object and real object. 

34. (Amended) The method according to claim 20, wherein one of the rules 
stored in the rule memory expresses the action of the virtual object as an action pattern with a 
predetermined aim for achieving the objective of the simulation . 



35. (Amended) The method according to claim 34, wherein said computation 
step determines an action using a [the] pattern having [has] a path disadvantageous to the 
operator [player] in consideration of a layout relationship between the virtual object and real 
object. 
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36. (Amended) The method according to claim 30, wherein when the operator 
[player] is one of real objects, said geometric information acquisition step uses an output from 
said viewpoint detection step of detecting the location/posture of the viewpoint of the operator 
that [player as information which is to be acquired in the geometric information acquisition step 
and] pertains to a location and shape of the operator [player]. 

37. (Amended) The method according to claim 20, wherein the viewpoint 
detection step includes the step of detecting a location/posture of the [a] head of the operator 
[player], and 

said method further comprises the detection step of detecting a location/posture of 
a hand of the operator [player]; and 

the step of recognizing a relative location of the hand of the operator [player] with 
respect to the head as a command on the basis of an output in the detection step. 

39. (Amended) A storage medium which stores a program of an image 
processing method for a simulator [game] apparatus with which an operator [a player] plays a 
simulation [game] with virtual object(s) in a mixed reality space including a virtual space and 
real space comprising: 

viewpoint detection program step for detecting the location/posture of a viewpoint 
of the operator [player]; 
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geometric information acquisition program step for acquiring geometric 
information of real object(s); 

recognition program step for recognizing a current[,] relative relationship between 
the virtual object(s) and real object(s); 

a rule memory for storing rules for controlling the action of the virtual object(s); 

computation program step for detemiining the next action of the virtual object(s) 
in accordance with the rules stored in said rule memory and in correspondence with the 
location/posture of the real object(s), and computing the [thee] location/posture of the virtual 
object(s) after the determined action; and 

presentation program step for generating at least one [of] image of the virtual 
object on the basis of the location/posture of the virtual object(s) after the action and the 
location/posture of the viewpoint position of the operator [player], and for representing the mixed 
reality space to the operator [player] by superimposing the image(s) of the virtual object(s} 
[image(s)] on the operator's [player's] view of the real space. 
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